Intelligent defect detection from image data

ABSTRACT

Implementations include receiving image data representative of images of items within a physical environment and depicting defects in at least one item, providing one or more of a set of augmented images using image augmentation based on the image data and a set of synthetic images using ML-based image synthesis based on the image data, processing one of the set of augmented images and the set of synthetic images using an ML model to provide a set of defect characteristics representative of defects in the at least one item, providing one or more root causes of each of the one or more defects by processing the set of defect characteristics and ancillary data, the ancillary data representative of the physical environment, and generating one or more alerts based on the one or more root causes for remediation of at least one root cause of the one or more defects.

BACKGROUND

Physical structures can require inspection for defects that can renderthe physical structure unfit for its intended purpose. In someinstances, failure of a physical structure due to a defect can result inlosses (e.g., financial losses, physical losses). Consequently,inspection systems have been developed in an effort to enhanceefficiencies in defect detection. In some examples, traditionalinspection systems process images of physical structures in an effort todetect occurrences of defects.

Some traditional systems implement rule-based object detection to detectdefects. However, rule-based detection struggles to adapt tomisalignment and tiny shifts in defect images. Some traditional systemsuse machine learning (ML) models to detect defects in images. However,such ML models need to be trained and training can require training dataincluding labeled defect images. Labeling of defect images typicallyrequires human experts to view and label each image, which is expensiveand labor-intensive. Also, the defect images are often imbalanced,meaning available defect image data is not uniformly distributed betweenthe different image classes (usually a strong minority class). Thisimbalance can negatively affect the accuracy of the resulting ML model(e.g., convolutional neural network (CNN) classifier).

Further, in order to remediate defects, causes of defects also need tobe determined. In some scenarios, a cause of a defect may need to bedetermined rapidly (e.g., in real-time, or near real-time) to avoidfurther occurrences of the defect and/or imminent failure. In somescenarios, a difference between a target structure (also referred to asgolden structure) and the actual structure should be determined rapidly(e.g., in real-time, or near real-time) to detect so-called killerdefects. Further, to address causes of defects, defect types, counts,locations, and severity are to be determined and appropriate personnelnotified to address the issues identified. Traditional defect detectionsystems are deficient in addressing one or more of these issues.

SUMMARY

Implementations of the present disclosure are generally directed to anintelligent defect detection (IDD) platform for detecting defects inphysical structures based on image data. More particularly,implementations of the present disclosure are directed to an IDDplatform that uses one or more of classic image augmentation and machinelearning (ML)-based image augmentation to detect and classify defects,determine characteristics of the defects, diagnose causes of thedefects, and to alert and deploy personnel to remediate the causes.

In some implementations, actions include receiving image datarepresentative of images of one or more items within a physicalenvironment, one or more images depicting defects in at least one item,providing one or more of a set of augmented images using imageaugmentation based on the image data and a set of synthetic images usingML-based image synthesis based on the image data, processing one of theset of augmented images and the set of synthetic images using one ormore ML models to provide a set of defect characteristics, the set ofdefect characteristics representative of one or more defects in the atleast one item, providing one or more root causes of each of the one ormore defects by processing the set of defect characteristics andancillary data, the ancillary data representative of the physicalenvironment, and generating one or more alerts based on the one or moreroot causes for remediation of at least one root cause of the one ormore defects. Other implementations of this aspect include correspondingsystems, apparatus, and computer programs, configured to perform theactions of the methods, encoded on computer storage devices.

These and other implementations can each optionally include one or moreof the following features: providing a set of synthetic images usingML-based image synthesis based on the image data includes processing theimage data using a generative adversarial network (GAN) that generates asynthetic image by injecting noise into the one or more images; imageaugmentation includes applying one or more of rotation, shifting,shearing, mirroring, flipping, translations, stretching, cropping,affine transformations, and scaling to the one or more images; the setof defect characteristics includes one or more of a defect count and adefect location; each of the defect count and the defect location areprovided using a regional proposals network (RPN); the set of defectcharacteristics includes one or more of a killer defect and adifference; the killer defect is identified based on applyingreinforcement learning (RL) zoom and RL-enhanced refinement to at leastone image; the difference is identified using reinforcement learning(RL) zoom and RL-enhanced refinement to at least one image, and byapplying image alignment between the at least one image and a goldenimage; the set of defect characteristics includes a defect type; thedefect type is determined based on a defect signature provided throughsemantic segmentation applied to at least one image; and the itemsinclude semiconductor wafers.

The present disclosure also provides a computer-readable storage mediumcoupled to one or more processors and having instructions stored thereonwhich, when executed by the one or more processors, cause the one ormore processors to perform operations in accordance with implementationsof the methods provided herein.

The present disclosure further provides a system for implementing themethods provided herein. The system includes one or more processors, anda computer-readable storage medium coupled to the one or more processorshaving instructions stored thereon which, when executed by the one ormore processors, cause the one or more processors to perform operationsin accordance with implementations of the methods provided herein.

It is appreciated that methods in accordance with the present disclosurecan include any combination of the aspects and features describedherein. That is, methods in accordance with the present disclosure arenot limited to the combinations of aspects and features specificallydescribed herein, but also include any combination of the aspects andfeatures provided.

The details of one or more implementations of the present disclosure areset forth in the accompanying drawings and the description below. Otherfeatures and advantages of the present disclosure will be apparent fromthe description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 depicts an example system that can execute implementations of thepresent disclosure.

FIG. 2 depicts an example conceptual architecture in accordance withimplementations of the present disclosure.

FIG. 3 depicts an example system architecture in accordance withimplementations of the present disclosure.

FIG. 4 depicts an example process that can be executed inimplementations of the present disclosure.

DETAILED DESCRIPTION

Implementations of the present disclosure are generally directed to anintelligent defect detection (IDD) platform for detecting defects inphysical structures based on image data. More particularly,implementations of the present disclosure are directed to an IDDplatform that uses one or more of classic image augmentation and machinelearning (ML)-based image augmentation to detect and classify defects,determine characteristics of the defects, diagnose causes of thedefects, and to alert and deploy personnel to remediate the causes. Insome implementations, actions include receiving image datarepresentative of images of one or more items within a physicalenvironment, one or more images depicting defects in at least one item,providing one or more of a set of augmented images using imageaugmentation based on the image data and a set of synthetic images usingML-based image synthesis based on the image data, processing one of theset of augmented images and the set of synthetic images using one ormore ML models to provide a set of defect characteristics, the set ofdefect characteristics representative of one or more defects in the atleast one item, providing one or more root causes of each of the one ormore defects by processing the set of defect characteristics andancillary data, the ancillary data representative of the physicalenvironment, and generating one or more alerts based on the one or moreroot causes for remediation of at least one root cause of the one ormore defects.

To provide further context for implementations of the presentdisclosure, and as introduced above, physical structures can requireinspection for defects that can render the physical structure unfit forits intended purpose. In some instances, failure of a physical structuredue to a defect can result in losses (e.g., financial losses, physicallosses). Consequently, inspection systems have been developed in aneffort to enhance efficiencies in defect detection. In some examples,traditional inspection systems process images of physical structures inan effort to detect occurrences of defects.

Some traditional systems implement rule-based object detection to detectdefects. However, rule-based defect detection is unable to adapt tomisalignment and tiny shifts in defect images. Some traditional systemuse ML models to detect defects in images. However, such ML models needto be trained and training can require training data including labeleddefect images. Labeling of defect images typically requires humanexperts to view and label each image, which is expensive andlabor-intensive. Also, the defect images are often imbalanced, meaningavailable defect image data are not uniformly distributed between thedifferent image classes (usually a strong minority class). Thisimbalance can negatively affect the accuracy of the resulting ML model(e.g., convolutional neural network (CNN) classifier).

Further, in order to remediate defects, causes of defects also need tobe determined. In some scenarios, a cause of a defect may need to bedetermined rapidly (e.g., in real-time, or near real-time) to avoidfurther occurrences of the defect and/or failure. In some scenarios, adifference between a target structure (also referred to as goldenstructure) and the actual structure to detect so-called killer defectsshould be determined rapidly (e.g., in real-time, or near real-time) toreduce quality risks and/or failure. Further, to address causes ofdefects, defect types, counts, locations, and severity are to bedetermined and appropriate personnel to address the issues identified.Traditional defect detection systems are deficient in addressing one ormore of these issues.

In view of the above context, implementations of the present disclosureare directed to an IDD platform that uses one or more of classic imageaugmentation and machine learning ML-based image augmentation to detectand classify defects, determine characteristics of the defects, diagnosecauses of the defects, and to alert and deploy personnel to remediatethe causes. As described in further detail herein, the IDD platform ofthe present disclosure includes a generative adversarial network (GAN)image synthetic pipeline to generate the labeled defect images forimbalance defect classes. The IDD platform also provides comprehensivedefect detection from image data using a computer vision (CV) deep CNNmodel and a reinforcement learning (RL) model. The IDD platform enablesflexible selection of common CV tasks for image classification, objectlocalization, and semantic segmentation. Further, the IDD platformprovides for RL-enhanced region proposals using a RL-mask region-CNN(RCNN) that provides optimized performance for object detection tospot-the-difference, determine defect counts and defect locations, andto identify killer defects. In some examples, the RL-enhanced approachof the present disclosure enables achievement of multiple orders ofmagnitude over traditional systems in terms of speed, efficiency, andaccuracy. As also described in further detail herein, the IDD platformapplies failure mode and effect analysis (FMEA) with supervised andun-supervised ML models to correlate defects information to causes(e.g., process, tool).

In accordance with implementations of the present disclosure, the IDDplatform automates an end-to-end process for smart manufacturing defectdetection and enhances performance (e.g., in terms of accuracy andtechnical resources expended) over traditional systems. The IDD platformutilizes one or more servers (graphics processing unit (GPU) servers),and big data and deep learning tool sets. The IDD platform performshyperparameter auto-tuning for the selected neutral network (e.g.,learning rate, batch size), and provides options for transfer learningon various deep learning models to reduce model training time. The IDDplatform also provides easy-to-use control panels and applicationprogramming interfaces (APIs) to manage cloud instances.

Implementations of the present disclosure are described in furtherdetail herein with reference to an example use case. The example usecase includes defect detection in semiconductor wafers within asemiconductor fabrication plant (semiconductor fab). In the examplecontext, and among other issues, manufacturers seek to enhance defectpattern classification and killer defect detection to reduce qualityrisks and quickly identify and remediate causes of the defects. Exampledefects can include, without limitation, scratches, cracking, andpattern defects. Example causes can include, without limitation, adefective photoresist spray nozzle, a defective chemical-mechanicalplanarization (CMP) pad, a defective furnace thermal coupler, andpresence of foreign particles. It is contemplated, however, thatimplementations of the present disclosure can be realized in anyappropriate use case. For example, and without limitation,implementations of the present disclosure can be realized in pipelineinspection (e.g., oil pipelines, water pipelines). In some examples,implementations of the present disclosure can be used to detect defectsalong a pipeline and prioritize areas of pipelines for directassessment.

FIG. 1 depicts an example system 100 that can execute implementations ofthe present disclosure. The example system 100 includes a computingdevice 102, a back-end system 106, and a network 110. In some examples,the network 110 includes a local area network (LAN), wide area network(WAN), the Internet, or a combination thereof, and connects web sites,devices (e.g., the computing device 102), and back-end systems (e.g.,the back-end (multi GPU server) system 106). In some examples, thenetwork 110 can be accessed over a wired and/or a wirelesscommunications link. For example, mobile computing devices, such assmartphones can utilize a cellular network to access the network 110.

In some examples, the computing device 102 can include any appropriatetype of computing device such as a desktop computer, a laptop computer,a handheld computer, a tablet computer, a personal digital assistant(PDA), a cellular telephone, a network appliance, a camera, a smartphone, an enhanced general packet radio service (EGPRS) mobile phone, amedia player, a navigation device, an email device, a game console, oran appropriate combination of any two or more of these devices or otherdata processing devices.

In the depicted example, the back-end system 106 includes at least oneserver system 112, and data store 114 (e.g., database). In someexamples, at least one server system 112 hosts one or morecomputer-implemented services that users can interact with usingcomputing devices. For example, the server system 112 of the back-end(multi GPU server) system 106 can host an IDD platform in accordancewith implementations of the present disclosure.

FIG. 2 depicts an example conceptual architecture 200 in accordance withimplementations of the present disclosure. In the example of FIG. 2, theexample conceptual architecture 200 includes an IDD platform 202 and asource system 204. In some examples, the source system 204 providesimage data to the IDD platform 202, which processes the image data toprovide real-time, or near-real-time defect detection, causedetermination, and initiating remedial action, as described in furtherdetail herein. In the depicted example, the IDD platform 202 includesone or more servers 206, classic image augmentation 208, ML-based imageaugmentation 210, an IDD engine (IDDE) 212, a defect characteristicsdetermination 214, a diagnosis system 216, and an issue routing and taskscheduler 218.

In the depicted example, the source system 204 includes image data 220,ancillary data 222, and a datastore 224. In some examples, the sourcesystem 204 represents an environment, in which the image data 220 isgenerated. For example, and in accordance with the example context, thesource system 204 can include a semiconductor fab, in whichsemi-conductor wafer are manufactured. In this context, the image data220 can include images of semiconductor wafers that have beenmanufactured. For example, the image data 220 can be generated using oneor more cameras that capture images of the semiconductor wafers. In someexamples, the ancillary data 222 includes information relevant to thesource system 204. In the example context, information relevant to thesource system 204 can include, without limitation, tool information(e.g., information on fabrication machinery), product information (e.g.,specifications of the semiconductor wafers that are manufactured), andexpert knowledge (e.g., typical defects and respective causes thereof).

In accordance with implementations of the present disclosure, and asdescribed in further detail herein (e.g., with reference to FIG. 3) theIDD platform 202 enables detection of defects, defect signatures, killerdefects, defect counts, and defect locations, and provides insights,such as root cause analysis (RCA) for remediation. The IDD platform 202is provided as an automated deep learning platform that automatessynthesis process (augmentation) of the image data 220 and CV tasks, andprovides driverless effort to identify defects and determine rootcauses. In some implementations, data stored in the datastore 224 (e.g.,the image data 220, the ancillary data 222) is ingested into the IDDplatform 202 through the one or more servers 206. In some examples, atleast one server 206 includes multiple-GPUs.

As described herein, real-world conditions can contain more variationthan a sample set. In the context of the present disclosure, the actualcondition of semiconductors manufactured in a semiconductor fab cancontain more variation (e.g., in defects) than is represented in sampleimage data, which is used to train a ML model. Consequently, a resultingML model would not generalize well (i.e., is deficient in accounting forvariation). Further, the size of training data (e.g., images) for rareor sensitive data (e.g., manufacturing defects) may be insufficient fortraining highly complex ML models (e.g., deep NNs). For supervisedtraining, labeling a large amount of image data with groundtruthinformation (e.g., semantic labels, bounding boxes) is time- andresource-consuming and expensive. Accordingly, training data can includevariation deficiency, volume deficiency, and/or labelling deficiency).

In view of this, and as depicted in FIG. 2, the IDD platform of thepresent disclosure provides the classic image augmentation 208 and theML-based image augmentation 210. In some examples, the classic imageaugmentation 208 provides one or more augmented images by applying oneor more augmentation techniques. Example techniques can include, withoutlimitation, rotation, shifting, shearing, mirroring, flipping,translations, stretching, cropping, affine transformations, and scaling,one or more of which can be random. In general, each augmented image canbe described as an altered version of an image.

In some examples, the ML-based image augmentation 210 synthesizesimbalanced, rare and expensive label images using one or more GANs. Insome implementations, a complex GAN is used, such as a BourGAN, whichgenerates augmented images that are visually more implausible incomparison to other types of networks. In some examples, a GAN can bedescribed as a deep neural network architecture that includes multiplenetworks (e.g., a discriminative network (discriminator), a generativenetwork (generator)) that work against one another (adversarial) toproduce some output. In general, the generator provides a syntheticimage by injecting noise into an image, and the discriminator attemptsto distinguish images as either actual or synthetic. As the networks aresimultaneously trained (in an unsupervised manner), the generatorimproves in producing synthetic images that are more and more realistic,and the discriminator improves in identifying synthetic image. In thecontext of the instant disclosure, the output of the ML-based imageaugmentation includes synthesized images, such as synthesized defectimages. In short, the ML-based image augmentation 210 enables time- andresource-efficient generation of labeled images (e.g., forminority-classes) to provide balance in a resulting CNN classifier, forexample. That is, imbalance would otherwise result from imbalanceddefect images (i.e., defect images that are not uniformly distributedbetween different image classes (usually a strong minority-class)).

In some implementations, a set of images, a set of augmented images,and/or a set of synthetic images are provided to the IDDE 212. Forexample, and as discussed herein, each of the set of augmented imagesand the set of synthetic images can be generated based on the set ofimages (e.g., by classic image augmentation and ML-based imageaugmentation, respectively). In accordance with implementations of thepresent disclosure, the IDDE 212 automatically applies multiple CV tasksusing deep learning models. Example tasks include, without limitation,semantic segmentation 230, object detection 232, and classification 234.The IDDE 212 further provides RL-enhancements for the object detection232. Example RL-enhancements include RL-enhanced zoom actions (e.g.,zoom in/out of images) and RL-enhanced refinement (e.g., imagerefinement).

In some examples, semantic segmentation 230 includes classifying eachpixel (or a group of pixels) in an image from a predefined set ofclasses, each class representing an object (e.g., a wafer, asemiconductor). In effect, semantic segmentation 230 results in a maskover each object within an image. The semantic segmentation 230 uses aML-model (e.g., CNN) to perform the semantic segmentation. In accordancewith implementations of the present disclosure, the semanticsegmentation 230 enables all possible defects to be obtained and definedin terms of defect contours to diagnosis defect patterns based on defectsignatures, as described herein.

In some examples, the object detection 232 includes detecting thepresence of one or more objects within an image, an object beingindicated by a bounding box provided within the image. The objectdetection 232 uses a ML-model (e.g., CNN) to generate regions ofinterest (ROI) marked with bounding boxes, extract visual features foreach of bounding box, and determine whether an object is present basedon the visual features. In accordance with implementations of thepresent disclosure, the object detection 232 provides RL-enhancedobject/anomaly detection by applying RL on top of a region proposalsnetwork (e.g., R-CNN, fast R-CNN, mask R-CNN). In this manner, theobject detection 232 automates image alignment (registration) andconcatenating of a reference image (golden image) and target image(actual image) to perform spot-the-difference as the defect/anomaly.Further, search trajectories are enhanced based on rewards (localizeimprovement) (e.g., rather than a brute-force sliding window approach).In some examples, the classification 234 includes classifying objects(e.g., distinguishing objects) within an image (e.g., classifyingobjects as semiconductors within an image). The classification 234 usesa ML-model (e.g., CNN) to perform the object classification.

In some implementations, the defect characteristics determination 214applies multiple ML-models to provide values for each of a set of defectcharacteristics. Example characteristics include, without limitation,defect signature, defect count, defect location, killer defects,spot-the-difference, and defect type. In some implementations, thedefect characteristics are processed by the diagnosis system 216, whichperforms FMEA for RCA. In some examples, the issue routing and taskscheduler 218 automatically schedules tasks based on results (e.g.,defect pattern detection and classification) to direct the alertmessages to the appropriate response team, provide instruction formanual prevention, and/or schedule repair services.

In accordance with implementations of the present disclosure, IDDplatform uses one or more CNNs (e.g., VGG, ResidualNet, Inception) todetermine killer defect types and to derive relationships between thesource system (e.g., a tool in the semiconductor fab) through the defecttypes.

FIG. 3 depicts an example system 300 in accordance with implementationsof the present disclosure. The example system 300 supports an IDDplatform for detecting defects, determining causes of defects, andalerting and deploying personnel to remediate issues, as described indetail herein. As depicted in FIG. 3, the example system 300 includes acomputing system 302 and an on-premise system 304.

In some examples, the computing system 302 is programmed to receive data(e.g., image data) from various data sources (e.g., one or morecameras). The computing system 302 can include one or more computingdevices (e.g., servers, personal computing devices). For example, thecomputing system 302 can be a cloud computing system that uses anetworked collection of computing devices that enable computationalunits, which may be virtual computing devices that are operating acrossthe collection of computing devices, to be deployed and scaled asneeded. Other configurations are also possible, such as an on-premisecomputing system with GPU-acceleration to handle deep learning in atime- and resource-efficient manner.

The computing system 302 includes a data collection subsystem 306 thatis programmed to obtain and collect data from data sources that areexternal to the computing system 302, as described herein. For example,the data collection subsystem 306 can receive data from devices thattransmit data over one or more networks 308 (e.g., the Internet and/orother networks). The data collection subsystem 306 can accept data asbatch data and/or stream data to receive data in real-time (or nearreal-time), such as real-time image data.

The computing system 302 includes an IDDE 310 that is programmed toreceive data obtained by the data collection subsystem 306. For example,the IDDE 310 includes an image augmentation module 320, a RL module 322,a semantic segmentation module 324, a defect classification module 326,and a RCA module 328 each of which can be provided as one or morecomputer-executable programs executed by one or more computing devices.

In some examples, the IDD platform provides one or more wizards to guidethe user to choose an image augmentation technique (e.g., classic,ML-based). In some implementations, the IDDE 310 (e.g., the imageaugmentation module 320) examines input images to evaluate one or morecharacteristics and to make a recommendation as to augmentationtechnique based on the one or more characteristics. An examplecharacteristic includes, without limitation, tensor dimension (e.g.,one-dimensional tensor, two-dimensional tensor, and three-dimensionaltensor, as provided by TensorFlow). For example, if the tensor-dimensionis more than three (3) (e.g., RGB), use of ML-based image augmentationis recommended. In response to selection of the image augmentationtechnique, the image augmentation module 320 uses the classic or theML-based (e.g., GAN-based) image augmentation techniques to synthesisexpensive label defect images in order to reduce overfitting, improvethe classification accuracy (e.g., as described above with reference toFIG. 2). In some examples, both classic image augmentation and ML-basedimage augmentation can be performed. For example, each is performed andrespective set of images are provided (e.g., a set of augmented images(classic), and a set of synthetic images (ML-based)). In some examples,a set of images is selected for further processing (e.g., CV tasksdescribed herein).

To reduce the manual efforts, the IDD platform can include a web-basedinterface, within which image data can be input (e.g., dragged anddropped). In response to input of the image data, the IDD platformautomatically processes the image data, as described herein, to learnthe natural features of the image data, generate augmented and/orsynthesized image data, and use the generated images can be used indownstream CV tasks.

In some implementations, the IDD platform provides questions (e.g.,through the wizard) regarding downstream CV tasks that are to beperformed. For example, the IDD platform can inquire as to whetherdefect object detection is to be performed, whether defectclassification is to be performed, and/or whether semantic segmentationis to be performed. In some implementations, the IDD platform providesrecommendations on ML models that can be used. For example, arecommendation can be provided based on the images that are to beprocessed. In some examples, an imbalance in classes is determined fromthe image data and one or more ML models are recommended based on theimbalance. In some examples, for each recommended ML model, computingresources required to train and execute the ML model, and the expectedaccuracy and time to train the ML model can be provided.

In further detail, and in the example context of semiconductorfabrication, example image classes can include, without limitation,particle defect, fiber defect, scratch, water mark, pellicle broken,pellicle scratch, contaminants, and fingerprint. Accordingly, an imageclass corresponds to a type of defect depicted in images of the imageclass. Further, each type of defect results in a respective level ofdamage to the overall process. For example, and in the example context,the pellicle broken type of defect is rare and difficult to detect. Inthe event of a pellicle broken defect, a photo mask of the semiconductorfab equipment must be repaired or replaces, which can carry significantcost. On the other hand, other types of defects (e.g., water mark,particle defect) can be caused by the environment and the causes thereofcan be more easily and cost-effectively addressed.

In some implementations, the RL module 322 performs the defect objectdetection, which is also referred to as RL-enhanced object detectionherein. In some examples, RL-enhanced object detection is performedusing a network with multiple sequential layers. A first layer includesRL-based image alignment, in which images of like regions from differentimages are overlaid for analysis. The RL-based approach is more accurateand efficient in aligning images where noise and slight differences arepresent. A second layer is provided as a R-CNN (or faster R-CNN, maskR-CNN)-based defect detector that processes the aligned images toprovide difference representations, that can be used for time- andresource-efficient detection and classification of image differences.

In further detail, the RL module 322 executes RL techniques for imagealignment, which outperform other techniques, particularly in instanceswhere differences are subtle (e.g., in semiconductor fabrication wheredefects are very subtle). As compared to other image registrationtechniques, the RL-based approach of the present disclosure does notrequire manual input of control points. Instead, the RL-based approachautomatically identifies the best control points to align the referenceand target structure on the image data. Given reference structurecontrol points, the RL-based approach enables a top-down search strategyto narrow down the location for the target structure and align with thereference structure. For example, the RL module 322 includes an RL agentthat uses the deep Q-Learning method to find the localization policywith the reward using Intersection-over-Union (IoU) between the targetstructure area and the predicted bounding box for each state. Incomparison to computationally intensive sliding window approaches, theRL-based approach of the present disclosure increases accuracy foranomaly detection (spot-the-difference in structure) and significantlyspeeds up computation performance.

In some examples, the semantic segmentation module 324 performs semanticsegmentation to provide a defect contour (also referred to as defectblob) for each defect within an image. Each defect contour isrepresentative of a defect pattern (or defect signature) that can beidentified by the IDD platform.

In some examples, the defect classification module 326 automate imageclassification using a set of deep CNN models (e.g., VGG, ResidualNet,Inception). The defect classification module 326 can identify thepresence of killer defects, if any, and can assist in derivingrelationship between the source system (e.g., a tool in thesemiconductor fab) through the defect types.

In some examples, the RCA module 328 performs classification,correlation and/or RL-based action tracking approaches for causeinference, as described herein. More particularly, and based on theinformation (e.g., defect type) provided by other modules, the RCAmodule 328 infers the underlying causes for process/tool/partsabnormalities that would result in the identified defects. In furtherdetail, relationships between defect type and failure pattern/signatureare defined, and a ML-framework is provided that enables variableimportance scores to be attributed to input features in addition toFMEA. In some examples, influence maps are provided, which identify topinfluencers of defect information (defect counts, defect location,defect signature, detect types, killer defect pattern). ThisML-framework and influence maps are used by the RCA module to identifyroot cause factors, understand each root cause and its impact, andaggregate risk of delay and delay time. This information is used by theRCA module 328 to identify one or more corrective actions. For example,the RCA module 328 can maintain a table of corrective actions that canbe indexed by one or more root cause factors and/or defect types. Inthis manner, the one or more corrective actions can be determined basedon a query that includes one or more of a root cause factor and a defecttype.

In accordance with implementations of the present disclosure, the IDDE310 provides one or more data models 330 that can be used by a front-endsystem 332 and/or an inference engine 334. In some implementations, theinference engine 334 can receive data from the data collection subsystem306 and/or from the frontend system 332. In some examples, the data isdata, to which the one or more data models 330 are being applied. Insome examples, users can submit queries through the front-end system332, which queries are analyzed by the inference engine 334. In someexamples, the inference engine 334 may run a background process thatcontinuously evaluates data that is received through the data collectionsubsystem 306 and can provide an alert to one or more appropriateentities (e.g., devices, user accounts) regarding the problem andinferred solution (e.g., when an issue is identified that needs to beaddressed).

In some examples, the inference engine 334 can process specific queriesthat are received through the front-end system 332 on a demand basis(e.g., as queries are received from users). For example, the front-endsystem 332 can include a query engine that is programmed to receivequeries, process queries (e.g., textual analysis of queries, interactionwith the inference engine 334), and to provide results. For example, aclient computing device 340 of the on-premise system 304 can be used byusers to submit the queries over one or more networks 308 (e.g., theinternet, LANs, WANs, wireless networks, VPNs, combinations thereof).

For example, the IDD platform of the present disclosure providesML-based auto-alerting and repair scheduling module. In some examples,based on the result of the RCA, the IDD platform notifies one or morestakeholders about a defect, the cause of the defect, and a severitylevel of the defect. In some examples, the severity level can indicatean urgency at which the cause of the defect should be addressed (e.g.,to minimize losses from repeated defects resulting from the cause).Example severities can include, without limitation, high (e.g., repairimmediately), medium (e.g., repair within 24 hours), and low (e.g.,repair within 1 week). In some examples, the IDD platform notifies andschedules repair of the defect with appropriate personnel. For example,the IDD platform can issue an alert to maintenance personnel indicatingthe location of the cause (e.g., which machine within the semiconductorfab), the cause, the type of defect resulting from the cause, and theseverity. In some examples, ML models are used to classify the correctsolution for corrective actions, to classify the severity level, and tomatch the correct solution to the correct tool owner/technician with theseverity level. In some examples, a task scheduler and alerting messagedistribution API sends out the correct information to the target taskowner (e.g., maintenance personnel).

With regard to the on-premise system 304, the client computing devices340 can be any of a variety of appropriate computing devices, such aslaptops, desktop computers, mobile computing devices (e.g., smartphones,tablet computing devices, wearable computing devices), embeddedcomputing devices (e.g., diagnostic devices), and/or other appropriatecomputing devices. The client computing devices 340 can receive data,similar to the data that is provided to the data collection subsystem306, through a data gateway 342 through which the data can be routed toappropriate ones of the client computing devices 340 and/or to the datacollection subsystem 306. The data can be generated by various datasources, such as one or more cameras 344. The data gateway 342 can alsoallow for information stored in a database 346, such as image data thatmay be stored and distributed in batches, to be communicated to the datacollection subsystem 306 and/or the client computing devices 340. Thedatabase 346 can also include external data (e.g., ancillary datadescribed herein). The data gateway 342 can communicate with the datacollection subsystem 306 over the network 308.

FIG. 4 depicts an example process 400 that can be executed inimplementations of the present disclosure. In some examples, the exampleprocess 400 is provided using one or more computer-executable programsexecuted by one or more computing devices (e.g., the back-end system 106of FIG. 1).

Image data is received (402). It is determined whether ML-based imageaugmentation is to be used (404). If ML-based image augmentation is notto be used, classic image augmentation is performed (406). If ML-basedimage augmentation is to be used, ML-based image augmentation isperformed (408). In some implementations, both classic imageaugmentation and ML-based image augmentation are performed. It isdetermined whether defect object detection is to be performed (410). Ifdefect object detection is to be performed, regional proposals areprovided (412). Defect count and defect location are determined (414).RL-enhancements are provided (416). One or more killer defects, if any,are identified (418). Spot-the-difference is performed (420) and theexample process 400 loops back. If defect object detection is to beperformed or has already been performed, it is determined whethersemantic segmentation is to be performed (422). If semantic segmentationis to be performed, semantic segmentation is executed (424). If semanticsegmentation is not to be performed or semantic segmentation has beenexecuted, it is determined whether defect classification is to beperformed (426). If defect classification is to be performed, defectclassification is executed (428). If defect classification is not to beperformed or defect classification has been executed, RCA is performed(430). One or more remediations are dispatched (432).

Implementations of the present disclosure provide one or more technicaladvantages. For example, the IDD platform introduces a combined imageaugmentation and image synthetization approach to enable classicaugmentation and/or GAN model to synthesize images for imbalance andrare labeled defect images. For image synthesis, this auto-ML approachfor GAN model zoo is able to learn natural features of a dataset (set ofimages), automatically run a GAN model zoo, and select an optimized GANmodel to generate realistic, high-quality samples (set of syntheticimages) for downstream tasks. Further, execution of multi-GAN models isfully automated and a web-based interface is provided. Users candrag-and-drop image data, and the IDD platform automatically learns thenatural features of the input image data, generate new image datasimilar to the observed ones without coding. The IDD platformautomatically produces a dashboard to select the optimal GAN model fordefect detection and classification.

As another example, the IDD platform of the present disclosure providesvarious models to choose from depending on the user's tasks (e.g.,localization, segmentation, classification,). The IDD platform providesautomated hyper-parameter tuning and flexible transfer learningselections (e.g., weights, number of layers). Further, the stacking ofdeep RL layers and regional proposal networks (RPNs) enhance objectdefect detection and image anomaly detection. This hierarchical objectdetection presents a neutral network architecture that enables: stepwisedeformation of a bounding box in its size, position, and aspect ratio totightly fit the detected object, and automate image alignment(registration), concatenation of the golden image and target image, thenspot-the-difference to identify the defect/anomaly locations betweengolden image and target images using RPN models.

As another example, the IDD platform applies FMEA as input incombination with ML models to generate influence maps that identify tolikely influencers (causes) of current defect information . Further, RCAis used to determine next best action given defect pattern andcharacteristics, and one or more ML models classify and provide rankingsof anomaly tools/parts/processes that caused the defect. A feedbackmechanism to the tool module expert to adjust the false positive andfalse negative results. That is, the auto alerting and repair schedulingof the IDD platform is used to route alerts to the response team todetermine next best action (e.g., message, email alert, manualprevention, calling repair service) given defect root causes.

Implementations and all of the functional operations described in thisspecification may be realized in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Implementations may be realized asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable medium may be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “computing system” encompasses allapparatus, devices, and machines for processing data, including by wayof example a programmable processor, a computer, or multiple (CPU and/orGPU) processors or computers. The apparatus may include, in addition tohardware, code that creates an execution environment for the computerprogram in question (e.g., code) that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal (e.g., a machine-generated electrical,optical, or electromagnetic signal) that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) may be written in any appropriate form ofprogramming language, including compiled or interpreted languages, andit may be deployed in any appropriate form, including as a stand aloneprogram or as a module, component, subroutine, or other unit suitablefor use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program may bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub programs, orportions of code). A computer program may be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows may also be performedby, and apparatus may also be implemented as, special purpose logiccircuitry (e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit)).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any appropriate kind of digital computer.Generally, a processor will receive instructions and data from a readonly memory or a random access memory or both. Elements of a computercan include a processor for performing instructions and one or morememory devices for storing instructions and data. Generally, a computerwill also include, or be operatively coupled to receive data from ortransfer data to, or both, one or more mass storage devices for storingdata (e.g., magnetic, magneto optical disks, or optical disks). However,a computer need not have such devices. Moreover, a computer may beembedded in another device (e.g., a mobile telephone, a personal digitalassistant (PDA), a mobile audio player, a Global Positioning System(GPS) receiver). Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices (e.g., EPROM, EEPROM, and flash memory devices); magneticdisks (e.g., internal hard disks or removable disks); magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory may besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations may be realizedon a computer having a display device (e.g., a CRT (cathode ray tube),LCD (liquid crystal display), LED (light-emitting diode) monitor, fordisplaying information to the user and a keyboard and a pointing device(e.g., a mouse or a trackball), by which the user may provide input tothe computer. Other kinds of devices may be used to provide forinteraction with a user as well; for example, feedback provided to theuser may be any appropriate form of sensory feedback (e.g., visualfeedback, auditory feedback, or tactile feedback); and input from theuser may be received in any appropriate form, including acoustic,speech, or tactile input.

Implementations may be realized in a computing system that includes aback end component (e.g., as a data server), or that includes amiddleware component (e.g., an application server), or that includes afront end component (e.g., a client computer having a graphical userinterface or a Web browser through which a user may interact with animplementation), or any appropriate combination of one or more such backend, middleware, or front end components. The components of the systemmay be interconnected by any appropriate form or medium of digital datacommunication (e.g., a communication network). Examples of communicationnetworks include a local area network (“LAN”) and a wide area network(“WAN”) (e.g., the Internet).

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations. Certain features that are described in thisspecification in the context of separate implementations may also beimplemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation may also be implemented in multiple implementationsseparately or in any suitable sub-combination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination may in some cases be excised from the combination, and theclaimed combination may be directed to a sub-combination or variation ofa sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemsmay generally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved. Accordingly, other implementations are within the scope of thefollowing claims.

What is claimed is:
 1. A computer-implemented method for identifyingdefects and causes thereof in physical environments using a machinelearning (ML)-based intelligent defect detection (IDD) platform, themethod being executed by one or more processors and comprising:receiving image data representative of images of one or more itemswithin a physical environment, one or more images depicting defects inat least one item; providing one or more of a set of augmented imagesusing image augmentation based on the image data and a set of syntheticimages using ML-based image synthesis based on the image data;processing one of the set of augmented images and the set of syntheticimages using one or more ML models to provide a set of defectcharacteristics, the set of defect characteristics representative of oneor more defects in the at least one item; providing one or more rootcauses of each of the one or more defects by processing the set ofdefect characteristics and ancillary data, the ancillary datarepresentative of the physical environment; and generating one or morealerts based on the one or more root causes for remediation of at leastone root cause of the one or more defects.
 2. The method of claim 1,wherein providing a set of synthetic images using ML-based imagesynthesis based on the image data comprises processing the image datausing a generative adversarial network (GAN) that generates a syntheticimage by injecting noise into the one or more images.
 3. The method ofclaim 1, wherein image augmentation comprises applying one or more ofrotation, shifting, shearing, mirroring, flipping, translations,stretching, cropping, affine transformations, and scaling to the one ormore images.
 4. The method of claim 1, wherein the set of defectcharacteristics comprises one or more of a defect count and a defectlocation.
 5. The method of claim 4, wherein each of the defect count andthe defect location are provided using a regional proposals network(RPN).
 6. The method of claim 1, wherein the set of defectcharacteristics comprises one or more of a killer defect and adifference.
 7. The method of claim 6, wherein the killer defect isidentified based on applying reinforcement learning (RL) zoom andRL-enhanced refinement to at least one image.
 8. The method of claim 6,wherein the difference is identified using reinforcement learning (RL)zoom and RL-enhanced refinement to at least one image, and by applyingimage alignment between the at least one image and a golden image. 9.The method of claim 1, wherein the set of defect characteristicscomprises a defect type.
 10. The method of claim 9, wherein the defecttype is determined based on a defect signature provided through semanticsegmentation applied to at least one image.
 11. The method of claim 1,wherein the items comprise semiconductor wafers.
 12. A non-transitorycomputer-readable storage medium coupled to one or more processors andhaving instructions stored thereon which, when executed by the one ormore processors, cause the one or more processors to perform operationsfor identifying defects and causes thereof in physical environmentsusing a machine learning (ML)-based intelligent defect detection (IDD)platform, the operations comprising: receiving image data representativeof images of one or more items within a physical environment, one ormore images depicting defects in at least one item; providing one ormore of a set of augmented images using image augmentation based on theimage data and a set of synthetic images using ML-based image synthesisbased on the image data; processing one of the set of augmented imagesand the set of synthetic images using one or more ML models to provide aset of defect characteristics, the set of defect characteristicsrepresentative of one or more defects in the at least one item;providing one or more root causes of each of the one or more defects byprocessing the set of defect characteristics and ancillary data, theancillary data representative of the physical environment; andgenerating one or more alerts based on the one or more root causes forremediation of at least one root cause of the one or more defects. 13.The computer-readable storage medium of claim 12, wherein providing aset of synthetic images using ML-based image synthesis based on theimage data comprises processing the image data using a generativeadversarial network (GAN) that generates a synthetic image by injectingnoise into the one or more images.
 14. The computer-readable storagemedium of claim 12, wherein image augmentation comprises applying one ormore of rotation, shifting, shearing, mirroring, flipping, translations,stretching, cropping, affine transformations, and scaling to the one ormore images.
 15. The computer-readable storage medium of claim 12,wherein the set of defect characteristics comprises one or more of adefect count and a defect location.
 16. The computer-readable storagemedium of claim 15, wherein each of the defect count and the defectlocation are provided using a regional proposals network (RPN).
 17. Thecomputer-readable storage medium of claim 12, wherein the set of defectcharacteristics comprises one or more of a killer defect and adifference.
 18. The computer-readable storage medium of claim 17,wherein the killer defect is identified based on applying reinforcementlearning (RL) zoom and RL-enhanced refinement to at least one image. 19.The computer-readable storage medium of claim 17, wherein the differenceis identified using reinforcement learning (RL) zoom and RL-enhancedrefinement to at least one image, and by applying image alignmentbetween the at least one image and a golden image.
 20. Thecomputer-readable storage medium of claim 12, wherein the set of defectcharacteristics comprises a defect type.
 21. The computer-readablestorage medium of claim 20, wherein the defect type is determined basedon a defect signature provided through semantic segmentation applied toat least one image.
 22. The computer-readable storage medium of claim12, wherein the items comprise semiconductor wafers.
 23. A system,comprising: one or more processors; and a computer-readable storagedevice coupled to the one or more processors and having instructionsstored thereon which, when executed by the one or more processors, causethe one or more processors to perform operations for identifying defectsand causes thereof in physical environments using a machine learning(ML)-based intelligent defect detection (IDD) platform, the operationscomprising: receiving image data representative of images of one or moreitems within a physical environment, one or more images depictingdefects in at least one item; providing one or more of a set ofaugmented images using image augmentation based on the image data and aset of synthetic images using ML-based image synthesis based on theimage data; processing one of the set of augmented images and the set ofsynthetic images using one or more ML models to provide a set of defectcharacteristics, the set of defect characteristics representative of oneor more defects in the at least one item; providing one or more rootcauses of each of the one or more defects by processing the set ofdefect characteristics and ancillary data, the ancillary datarepresentative of the physical environment; and generating one or morealerts based on the one or more root causes for remediation of at leastone root cause of the one or more defects.
 24. The system of claim 23,wherein providing a set of synthetic images using ML-based imagesynthesis based on the image data comprises processing the image datausing a generative adversarial network (GAN) that generates a syntheticimage by injecting noise into the one or more images.
 25. The system ofclaim 23, wherein image augmentation comprises applying one or more ofrotation, shifting, shearing, mirroring, flipping, translations,stretching, cropping, affine transformations, and scaling to the one ormore images.
 26. The system of claim 23, wherein the set of defectcharacteristics comprises one or more of a defect count and a defectlocation.
 27. The system of claim 26, wherein each of the defect countand the defect location are provided using a regional proposals network(RPN).
 28. The system of claim 23, wherein the set of defectcharacteristics comprises one or more of a killer defect and adifference.
 29. The system of claim 28, wherein the killer defect isidentified based on applying reinforcement learning (RL) zoom andRL-enhanced refinement to at least one image.
 30. The system of claim28, wherein the difference is identified using reinforcement learning(RL) zoom and RL-enhanced refinement to at least one image, and byapplying image alignment between the at least one image and a goldenimage.
 31. The system of claim 23, wherein the set of defectcharacteristics comprises a defect type.
 32. The system of claim 31,wherein the defect type is determined based on a defect signatureprovided through semantic segmentation applied to at least one image.33. The system of claim 23, wherein the items comprise semiconductorwafers.